Ontology-based evaluation of organizational memory

Organizational Memory (OM) is regarded as an imperative implementation for organizations striving for innovation and competitive advantage. Yet, to date, there is no accepted and comprehensive methodology for developing OM. This study attempts to take a first step in this direction by concentrating on the evaluation of OM. We approach this challenge with ontology for the Organizational Memory (OM) domain, i.e., a schema (map) to guide SM design, construction, evolution and evaluation. The top level of the ontology is specified using the new generalized concept of Structural Memory (SM), which is a framework of type (e.g., Organizational Memory and Individual Memory) and component (content and mean) that enable management of knowledge and organizational learning. We present findings from empirical research to establish the claim for the ontology usability in facing the challenge of OM evaluation and design. This ontology is available for organizations maintaining OM as well as those who consider developing one. The potential benefits of this endeavor are with its theoretical grounds and with its empirical proof. The feasibility of our approach is demonstrated in a series of case studies conducted in various organizations in Israel and in Europe. Introduction Organizational Memory (OM) is critical for learning and communication in organizations (Walsh and Ungson, 1991; Te’eni, 2001). However, as evident from several recent reviews of OM and knowledge management, more generally, there is no theoretical foundation to adequately guide the development of OM (Alavi & Leidner, 2001). For one, implementations of OM vary considerably, see for example the different practices found in different industries and subject areas. Additionally, researchers have taken a piece-meal approach in studying and recommending OM development methodologies (Te’eni & Weinberger, 2000). For example, there have been extensive efforts at design (Abecker et al., 1998; Ackerman, 1998; Conklin, 1996; Dieng et al., 1999) but scarce attempts at developing methods for the analysis and evaluation of OM. Motivated by the need for a comprehensive view, we present a framework for the development and evaluation of OM based on domain ontology for OM. Using ontology one can conceptualize entities in some domain, and the relationships between them, to provide domain practitioners with a model of the domain (Chandrasekaran et al., 1999; Frank, 1999; Gruber, 1995). As the interest in implementing OM increased, design and evaluation remained challenges yet to be addressed. Particularly, the process of Organizational Memory evaluation is in need for objects, methods and means – as could be extracted from a grounded theory, represented by semi-formal domain ontology such as the one presented here. We claim that the ontology can be used generically, regardless of industry sector. Our use of the term “Organizational Memory” requires some explanation. The term OM is used for the subject domain. That is to be differentiated from the term Structural Memory that will be introduced hereafter. The ontology discussed here was developed in the frame of reference of a three-staged evaluation model, these stages are: 1. Development and evaluation of a normative Domain Ontology (DO). 2. Development and evaluation of OM ontologies developed in practice, denoted as Actual Ontologies. 3. Comparative evaluation of the DO with actual ontologies. Weinberger, Te’eni, Frank 2 A brief review of these stages will unfold by their order of appearance. As the ontology takes after the literature, knowledge acquisition was based on an exhaustive literature review. Evaluation was motivated at content and at design aspects. Verification of design was facilitated drawing on the OO paradigm, using Information System methods and ontology evaluation criteria. Two criteria were used for validation conceptual coverage and usability, the latter is expanded below. The creation of a DO is necessarily subjective (Fridman Noy & McGuiness, 2001); for this reason the importance of domain ontology evaluation is with its usability, hence the three-staged evaluation driven model. The claim for usability is established using two criteria: competence – established through the development of the ontologies for OM in existing organizations (as described in stage 2), and utility – established by the performance of quantitative and qualitative comparative evaluation (as described in stage 3). Comparative evaluation is required to support the contextualization and validation of conceptualization processes, bridging between two epistemological stances: the first is generic, theoretic, and interdisciplinary – gained through the study of the literature, and the second is social; an organizational epistemology – gained through the exploration of OM in organizations. Going through the description of the ontology development process is maintained using a set of criteria for conducting and evaluating interpretive field studies in IS suggested by Klein & Myers (1999). The comparative evaluation complements the process demonstrated by the evaluation model. The organization of this paper responds to the challenges noted above. The next section provides a description of the DO. We then describe the practice of ontology development and the comparative evaluation. We conclude with a discussion and recommendations for further research. Domain ontology Theoretically – “The subject of ontology is the study of the categories of things that exist in some domain” (Sowa, 2000); practically, ontology is an explicit specification of conceptualization intended for communication, sharing and reuse (Gruber, 2004). Ontology “necessarily entails or embodies some sort of world view with respect to a Weinberger, Te’eni, Frank 3 given domain” (Uschold & Gruninger, 1996) – especially as ontology development has moved from the artificial intelligence labs to domain experts’ practices (Fridman Noy & McGuinnes, 2001). However, ontology development is yet more a craft than a method (Dieng et al., 1998; Fernandez, 1999; O’Leary, 1998) and, moreover, the problem of ontology evaluation “is one of the most under-developed areas in ontology design” (Fridman Noy, 1997). The ontology described here acknowledges these observations. Following a comprehensive development process and an integrated approach taken at evaluation, it includes a certain worldview to establish its intended use as an OM evaluation tool, designed for human understanding. Development of the OM ontology Motivated by behaviorist perspective, Walsh & Ungson (1991) observed that: “Organizational memory is both an individual and organizational-level construct... Organizational memory refers to stored information from an organization history that can be brought to bear on present decisions” (p. 61). Several years later, Stein & Zwass (1995) brought up the IS dimension stating that OM is: “The means by which knowledge from the past is brought to bear on present activities, thus resulting in higher or lower levels of organizational effectiveness” (p. 18). Other theories have put forward knowledge and Meta knowledge (Te’eni & Schwartz, 1999) and knowledge creation processes (Nonaka & Takeuchi, 1995) – to name a few. The construction of a single ontological tree could therefore be infeasible. Considerations should be explored theoretically as demonstrated by the methods hereby described, and for practicality – as was approached using the examples described. All in all, conceptualization efforts at modeling existing theories were met by epistemological and methodological considerations. These approaches will be described and exemplified through the description of determining the hierarchy of upper-level concepts, to be described hereafter. Epistemological approach. The epistemological approach was established through contextual understanding of the domain. Considering that focus has been shifted from storage to active memory could result in mapping knowledge as leading concept. That would pave way to yet another dilemma – what do tacit and explicit count Weinberger, Te’eni, Frank 4 for: are these entities or attributes? Rather yet, realizing the balance existing between people – individuals and groups alike, to products – content materials and agents, conjoined the tendency to avoid multiple inheritances. For example, such as would result from mapping individual, knowledge or knowledge system to lead the ontology. The pitfalls of this attitude could result in a network of interwoven concepts, which would have dissolved the intention of getting a coherent structure. Methodological approach. The methodological approach entails four stages: (1) specification and Knowledge acquisition, (2) analysis and knowledge structuring, (3) documentation, and (4) evaluation. Each of these stages of developing the ontology involves several methods and techniques. The different methods helped us converge gradually and iteratively towards a more complete ontology. Below are several demonstrations of the methods we used. (1) The whole-part relationship (and the classified or non-classified concepts distinction) (Guarino et al., 1994; Sowa, 1995) has led us to distinguish between Individual Memory (IM) and Organizational Memory (OM) by including IM as part of the OM (these classes are expanded below). (2) Observing related terms (Fernandez et al., 1999) had led to identifying several common types of memory such as OM, corporate memory and enterprise memory. (3) context-specific analysis of concepts (Guarino, 1997), pointed at the difference between multiple (OM) and singular (IM), thus leading to the decision to distinguish between (4) abstract and concrete concepts and between objects and processes (Gruninger & Fox, 1995), These techniques resulted in the upper level of our proposed ontology, as shown in figure 1. Once the upper level was determined, we turned to (5) management of classifiers’ hierarchy (Fridman Noy & McGuiness, 2001) of the (relatively) lower level abstract classes and the concrete classes. Following the OO approach and UML conventions, we iteratively used (6) an analysis of hierarchical structures and (7) categorized information as entities and attributes. We finalized the development of the ontology by (8) assigning values to attributes – at the highest ontological level possible. Modeling and documentation relied on a combination of four techniques: rules engrained in the development tool language (i.e., UML), information systems development methods, ontology development methodologies and development heuristics, such as setting attributes’ value types. Development platform combined Weinberger, Te’eni, Frank 5 UML and the Rational Rose tool using UML notational conventions Class, Use case, Activity, and Sequence diagrams (Booch et al., 1999), supported by ontology development theories (Fensel, 2001). Using UML, which enables separate development of logical and use case view, further supported the definition of the two aspects, static and dynamic. To match design goals, (e.g., usability, human understanding) documentation followed double path, resulting in 47 tables of semi-formal verbal description and about 100 UML diagrams representing a bi-dimensional view: static – using class diagrams, and the dynamic – using use-case, activity and sequence diagrams. The resulting ontology Structural Memory. The upper-level concept of the ontology, Structural Memory (SM) is a new concept a bi-dimensional framework. “Structural Memory (SM) is an abstract classifier that defines a framework of types (e.g., Individual Memory and Organizational Memory) and components (Content & Mean) that enable management of knowledge and organizational learning (see and example in table 1). SM is described using two attributes: goal & strategy, and one method: SM Management. The SM class has two subclasses associated with it, Type and Component, using the aggregation (i.e., “part-of”) relation” (Weinberger, 2004). Of the two SM dimensions, the component classifier refers to content (knowledge and meta knowledge) and mean (agent and processes) and subclasses. The type classifier refers to Organizational Memory, Individual Memory and subclasses. Both classes are associated with SM using aggregation, to convey the meaning of unity, indispensability and vitality. The resulting model is depicted below (see figure 2). The two attributes assigned to SM are strategy and goal. The value defined for Strategy is recognizing diversities in motivations between organizations practicing OM. The same is true for goal. SM origin is with the observation that there is a need for an independent term to convey the meaning of unity of elements – people, instruments and processes that comprise an OM. The term OM is used as an umbrella term for two common situations: in the disciplinary aspect, much like KM, and conveys community meaning, in contrast to individual aspect. It is for this reason, that from ontological point of view, we found it beneficial to generalize the usual OM type, by defining the SM Weinberger, Te’eni, Frank 6 abstract type. The term OM is used in the ontology by its second common usage – indicating the plural form of the Type dimension. SM’s two dimensions – type and component are bound together by the materialization interface. In component dimension all concepts up to level four are abstract, the same is true up to level three for type dimension (see figure 2). Type. Type and subclasses represent rethinking existing models. Type identification enables the development of two different views of SM, underlying the importance of individual-organizational interaction as emphasized throughout the literature. The abstract (third) level represents the plural – OM, and the singular – IM forms. Using generalization relationship, for each there are concrete type forms: Personal Memory and Group Memory, Corporate Memory and Enterprise Memory, respectively. Methods defined for type – transfer, and for OM – include, integrate social mechanisms, as exemplified in the knowledge creation theory (Nonaka & Takeuchi, 1995). These methods are further detailed in SM lifecycle. Component. The design of the two aspects of component – content and mean, follows previous models. Distinctly it appears with Wijnhoven (1999) who identified two inter-related aspects of OM: content and means. There with a slightly different sub concept: for content it is there knowledge and information and for means it is processes and media. However, conceptualization of content here takes also after (Te’eni & Schwartz, 1999) who have put forward the importance of Meta knowledge; The definition of content class and its subclasses is motivated by the assertion that perceives knowledge resource interoperability – represented by an interface, dependent upon meta-knowledge constructs. Mean class is perceived as a composition of agents and processes – conceptually bound together through the performance interface (see figure 2). SM lifecycle. The only concept in the ontology to be represented twice is SM lifecycle. It is represented in the static aspect as a subclass of mean, and detailed in the dynamic aspect – using the use case view. SM lifecycle realizes an epistemological aspect – referring to knowledge creation processes, and ontological aspect – referring to specification and control processes. There are six stages in SM lifecycle, subdivided to planning actions – specification and analysis and synthesis, and development and Weinberger, Te’eni, Frank 7 control – design, construction, innovation and evaluation. Embodying SM lifecycle takes after classical IS development methodologies. It evolves dynamically, through a continuous iteration between classical Information systems – depicted in mean, and social systems – depicted in type. Baring in mind the intended usage of the ontology, this classification reflects understanding of SM nature – with regard to structural, procedural, and cognitive aspects. The ontology reflects a worldview that encompasses a multi-disciplinary view ranging through several aspects of knowledge systems theories: social–behaviorist, information science and information system driven. To conclude, we use a final note on the ontology evolution process. As in any conceptualization effort, evaluation was conducted all through development lifecycle for conceptual coverage and exhaustion (Gruber, 1995). Evaluation took after the two aspects of ontology development: content and design practicing verification and validation. Verification was performed using the criteria of clarity, coherence, Minimal Ontological bias & extendibility (Gomez-Perez, 1995; Gomez-Perez, 1998; Fridman-Noy & Hafner, 1997). Validation was performed for using the criteria of conceptual coverage and usability (Gomez-Perez & Feranadez, 1996; Gruninger & Fox, 1995; Fridman-Noy, 1997; Fridman Noy & Musen, 1999). Of these two, the later was carried out to establish feasibility, grounding theory through empirical research – which is to be described in the following section. Based on this unified perception, it is expected that drawing on the ontology will demonstrate competency – enabling development of common ontologies, and usability – answering for design and evaluation concerns of organizations planning to or already practicing organizational memories. Domain ontology in practice Governed by the need to diversity, we chose four organizations to participate in the empirical research. Diversity shows in their location – being in Europe and in Israel, and in their sector, being private – commercial (HCS, Amigo) and public – research oriented (DLR), and service oriented (LSV) organizations. The case study technique is practiced here alongside other techniques in an integrated approach at evaluation, Weinberger, Te’eni, Frank 8 designed to respond to the so-called lack of formality of domain ontology and its evaluation. In the realm of case study research, much of the effort in designing research has intrigued the development of theories intended at fostering the usage of communication methods to inform research design. One such example is a set of criteria for conducting and evaluating interpretive field studies in IS suggested by Klein & Myers (1999). The description of empirically utilizing the model unfolded here, will be annotated by the description of how it responds to these seven criteria: interaction between the researcher and the subjects, contextualization, abstraction and generalization, multiple interpretations, dialogical reasoning, suspicion, and the (first) principle of the hermeneutic circle – which will be referred to in the discussion that concludes this paper. Motivated by technicality we had to focus on selected SM types in each organization. However, from each organization there is a variety of types and the overall analysis is inclusive. Actual ontology development methodology was adapted from the DO development methodology. In order to retain understanding to the subject matter amongst participants, a preliminary presentation and discussion of the ontology was carried out – to bridge semantic gaps and to establish conceptual agreement. Special efforts were focused on knowledge acquisition: (1) the development and practice of questionnaires (i.e., using three techniques: Top-down – mapping upper-levels entities, middle-out – mapping middle levels entities and Bottom-up – mapping concrete levels entities), and (2) using the interview technique in parallel. In these cases, interviews support overcoming challenges resulting from possible further semantic and cultural barriers. Knowledge acquisition processes were followed by (2) analysis and documentation, and by (3) evaluation. To gain the benefits of the existing (i.e., DO) documentation, it was incorporated into the interview procedure to support responding to questionnaires, thus maintaining interaction between the researcher and the subjects, to obtain contextualization through data usage – encompassing social and historical background. Passing multiple questionnaires for each type explored, resulted in having several account to relay upon – thus following the principle of multiple interpretations. Integrating data obtained from several participants for the same type had the effect of Weinberger, Te’eni, Frank 9 abstraction and generalization towards establishing our view of the actual ontology. Going through these procedures it was observed that usability consideration, taken during development, have ensured the following: • Definition of abstract level concepts promotes understanding and agreement and thus identification and development of ontologies in practice. • Development of concrete levels facilitates data acquisition. • Documentation – verbal and visualized, facilitates mapping and development. • Conceptualization of a bi-dimensional view, using the type – component structure, facilitates understanding, contextualization and validation. Taken that ontologies find proof by answering for usability – as was also lately commented by (Gruber, 2004) we had reasons to believe that Ideally, organizations that apply an OM, would appreciate a feedback that would suggest theoretical and practical valuation of their design. To achieve this goal, we have taken to approach two aspects of utility evaluation: quantitative – comparative, based on four predefined upper-level concepts form the ontology, and qualitative – based on four rules which are part, though not formal, of the ontology. This is aimed at establishing the claim for usability. Taking this step at the evaluation of usability corresponds to the third and last stage of the evaluation model. Quantitative evaluation. Quantitative evaluation constitutes grounds for the next operation. It was manipulated to validate findings of common ontologies using a set of five theoretical questions, designed for IS evaluation (Zachman, 1995), that is: what – what upper-levels component dimension entities are present, who – who are the active type represented, how – how does its lifecycle evolve, and where – on what levels are core phenomena represented. Qualitative evaluation. Qualitative evaluation was conducted to respond to the question: “is the domain ontology useful?” – that is, further to mapping the existing situation, can it point at of the implications of shortages identified in the existing situation Weinberger, Te’eni, Frank 10 in order to inform further design. Qualitative evaluation was conducted through an integrative approach taken at all ontologies modeled for a specific organization. It followed rules prescribed by the ontology that demonstrate the interplay between upperlevel (abstract) and lower-level (concrete) entities in the ontology, between dimensions and between entities and attributes assigned to entities. These rules are depicted below. To demonstrate the practicality of the rules and the feasibility of our approach, a detailed discussion, integrating findings from all four case studies is unfolded hereafter, In the order of appearance of the four rules. Structural Memory. Four conditions are mentioned in the SM rule. We will describe the findings obtained taking after their order of appearance. Three organizations, CHS, LSV and Amigo, have their SM only partially represented, since they do not have IM. For each one of these three, there is only partial representation of component space. Each organization was found to be operating to fulfill its strategy: CHS was motivated at promoting usage of exiting resources and creating cultural terms for sharing, LSV was engaged in assimilating change, DLR was found to be involved with proceeding innovation, and Amigo engaged in collecting and creating content resources. The key for understanding these findings is with organizational culture and constraints – attributes assigned to OM. Type. The type rule, per-se, is only maintained in DLR, where all type kinds exist and maintain aggregation relation. In CHS there is an occasional aggregation between GM1 and EM1– else than that, all SMs demonstrate association relation only. In several organizations we have found dominance of one SM type, rather than sharing and integration amongst types. For example, EM1 in CHS maintains a battery of components – knowledge and agents, available for other SMs, but not vice versa. In both CHS and in LSV there are CMs who, for absence of means, only maintain association relation with other SMs. In Amigo, shortage in SMs rather leaves GM1 to be operating on its own terms. In CHS, we have found several CMs that continuously create knowledge; alas they miss the adequate means to practice sharing through transfer-inclusion materialization, as envisioned. In these cases, relationships between SMs were found to association (rather than aggregation). Weinberger, Te’eni, Frank 11 Component. The interrelations prescribed by the rules are evident with all four organizations. For example, EM1 in CHS could share its knowledge using its agents, while LSV was unable to share its knowledge for lack of agents. Drawing upon its strategy and constraints, CHS has an extensive component environment for EM1, though not for either of its CMs. Consequently, sharing one agent between SMs (KM technology) results in sharing also the SM manger agent. Considering the envisioned interrelations between content and mean subclasses, findings indicate that their existence, such as in DLR, or non existence, such as in LSV, effects also the relationships between SMs (i.e., sharing patterns). SM lifecycle. Management of SM lifecycle is defined as a method for SM. There is an agent assigned for this and its dynamic and iterative course unfolds in the use case view. As its materialization is dependent upon representation of both types – IM and OM, it is only fully exploited in DLR. However, development pattern was tracked in SMs maintaining partial representation. In CHS, LSV and Amigo, there was a tendency towards construction – implementation and use, and towards evolution – maintenance and growth, regardless of analysis and synthesis or evaluation. Table 4. Ontology rules Rule Description Structural Memory Complete representation includes at least one of each of SM concrete level types. For each, there should exist representation of the components dimension. All types should operate towards the fulfillment of predefined goals acting by predefined strategy. Type Complete representation of types space should include aggregation relationships between types on the concrete levels. That is to ensure knowledge sharing processes within the materialization of complete SM lifecycle. Component Materialization of the component dimension will include representation of KR & MKR (the latest is responsible for the first interoperability) as well as Process and Agent when the latest is responsible for the execution of the first. SM Lifecycle Complete lifecycle representation will include representation of both attitudes towards SM: IM and OM when one is aggregated into the other. Weinberger, Te’eni, Frank 12 Applying quantitative and qualitative methods for utility evaluation, results in understanding the meaning and the significance of the results obtained, thus producing added value for participating organizations. Going through ontology development and evaluation responds to the principle of dialogical reasoning – which unfolds as findings from ontologies in practice are extracted to review previous assumptions (i.e., incorporated in the domain ontology). The flow of events for ontology development by organizations; constructing an organizational view via modeling several SMs while having several people respond to the same SM also responds to the (last) principle of suspicion. In consequence, data interpretation is used to show how (so-called) theoretical principles are inscribed into the field details, and how confronting the two sorts contribute to the definition of one and the evolution of the other. All considered, the claim for utility is based on a two-aspect view: that which derives from the competence to model actual ontologies based on the DO, and that which derives from the qualitative and qualitative evaluation of the ontologies in practice. The methodology for development and evaluation of a case study, as demonstrated here is semi-formal; therefore it can be repeated on demand.